Path Side Image in Map Overlay

ABSTRACT

One or more systems, devices, and/or methods for generating a map including path side data include storing path side data referenced to three-dimensional geographic coordinates. The path side data may be optical data or optical data modified based on one or more panoramic images. The path side data is combined with map data received from a map database. The map data includes nodes and segments. A processor rotates the path side data based on one of the segments. The rotation may be about the segment or about a featured identified in the optical data. The path side data overlaid on the map data is outputted to a display, a file, or another device.

REFERENCE TO RELATED APPLICATIONS

The present patent application is related to the copending patentapplication filed on the same date, Ser. No. 13/340,923, entitled “PATHSIDE IMAGERY,” Attorney Docket No. N0408US, the entire disclosure ofwhich is incorporated by reference herein.

BACKGROUND OF THE INVENTION

The present invention relates to maps and navigation and, moreparticularly, to method(s) and system(s) for overlay path side imageryon a map.

Conventional map applications include graphical representations of roadsas well as icons corresponding to points of interest. The mapapplications may calculate and display a route from an origin point to adestination point. The route may be selectable between driving, walking,public transportation, and biking.

Some map applications illustrate a view from the perspective of aroadway through the manipulation of image bubbles. Image bubbles arepanoramic images collected with respect to a single perspective. Theimage bubbles may extend nearly 360 degrees along one or more axes.Zooming and panning within an image bubble provides a view from aparticular viewpoint. To view another perspective, a typical streetlevel viewing application must be switched to the next image bubble.That is, in order to view the scene along the side of the street, a usermust select a direction to move to the next image bubble and wait forthe next image bubble to load, which may also require a panningoperation.

SUMMARY OF THE INVENTION

One or more systems, devices, and/or methods for generating a mapincluding path side data are described. The path side data is storedreferenced to three-dimensional geographic coordinates. The path sidedata may be optical data or optical data modified based on one or morepanoramic images. The path side data is combined with map data receivedfrom a map database. The map data includes nodes and segments. Aprocessor rotates the path side data based on one of the segments. Therotation may be about the segment or about a featured identified in theoptical data. The path side data overlaid on the map data is outputtedto a display, a file, or another device.

The present invention is defined by the following claims, and nothing inthis section should be taken as a limitation on those claims. Furtheraspects and advantages of the invention are discussed below inconjunction with the preferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for generating a map including path sideimagery.

FIG. 2 illustrates the projection of optical data on a two-dimensionalplane.

FIG. 3 illustrates a view correlating the optical data to an imagebubble.

FIG. 4 illustrates another view correlating the optical data to theimage bubble.

FIG. 5 illustrates mapping the optical data to an image bubble.

FIG. 6 illustrates a map including orthographic path side imagery.

FIGS. 7A and 7B illustrate a rotation of path side data.

FIG. 8 illustrates another map including path side imagery.

FIG. 9 illustrates a detailed view of the server of FIG. 1.

FIG. 10 illustrates a detailed view of the user device of FIG. 1.

FIG. 11 illustrates a map including silhouettes of path side imagery.

FIG. 12 illustrates a map including outlines of path side imagery.

FIG. 13 illustrates a map including optical data of path side imagery.

FIG. 14 illustrates a map including a perspective view of path sideimagery.

FIG. 15 illustrates the projection of optical data on layeredtwo-dimensional plane.

FIG. 16 illustrates a plurality of image bubbles correlated with pathside imagery.

FIG. 17 illustrates an example obstruction using a plurality of imagebubbles.

FIG. 18 illustrates a flowchart for generating a map including path sideimagery.

FIG. 19 illustrates curved projection planes.

FIG. 20 illustrates an intermediate projection surface.

FIGS. 21A and 21B illustrated another embodiment for generating a mapincluding path side imagery.

DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS

FIG. 1 illustrates a system 150 for generating a map including path sidedata or imagery. The system 150 includes a user device 100, a network110, a server 120, and a geographic and/or map database 130. Thegeographic database 130 may be maintained by a map developer, such asNAVTEQ North America, LLC located in Chicago, Ill. The map developer maycollect geographic data to generate and enhance the geographic database130. The user device 100 may be a cellular telephone (smart phone), apersonal digital assistant (“PDA”), a tablet computer, a laptop, apersonal navigation device (“PND”), a personal computer or anothercomputing device.

The system 150 receives path side data referenced to three-dimensionalgeographic coordinates, which is stored in the database 130. Thedatabase 130 or another location stores map data. The map data includesa plurality of nodes and a plurality of segments. The nodes representintersections or intersection points of links or path segments, and thelinks or path segments represent any traversable path. For example, eachlink or path segment may correspond to a street, a road, a pathway, apedestrian walkway, a train path, a waterway, or any other navigablepathway. The system 150 selects a segment from the plurality of segmentsin the map data. The segment may be selected based on a location of theuser device 100. The segment may be selected based on a command from auser.

The system 150 may be configured to receive optical data. The opticaldata may be a depthmap generated from an optical distancing system. Theoptical distancing system may be a light detection and ranging (LIDAR)device, a stereoscopic camera, or a structured light device. The opticaldata may be created using any arbitrary viewpoint or perspective. Theoptical data may be generated for each panoramic image viewpoint andstored in the database 130. The optical data may include geo-coordinatessuch as latitude, longitude, and altitude for each of plurality ofpoints.

The system 150 projects the optical data onto at least onetwo-dimensional plane. The two-dimensional plane may be parallel to thestreet, parallel to the associated path segment, and/or parallel to thedirection of movement of the optical distancing system as the opticaldata is collected. The two-dimensional plane may be flat, curved, orparametric. The two-dimensional plane may be selected to estimate abuilding facade along the street.

The system 150 receives image data from at least one panoramic image orimage bubble. The at least one image bubble may be collected by acamera. The image bubble may have a center point measured in Cartesiancoordinates such as an X-coordinate, a Y-coordinate, and a Z-coordinate.Each point on the image bubble is defined by the center point and one ormore angles (e.g., roll, pitch, yaw). By correlating the Cartesian spaceof the image bubble and the geo-referenced three-dimensional space ofthe optical data, the system 150 associates one or more points of theoptical data with one or more pixels in the image bubble.

The two-dimensional plane is modified to illustrate path side imagery.Specifically, the pixel values of the image bubble are applied to theoptical data projected onto the two-dimensional plane. In other words,the system 150 calculates a pixel value for each point of the opticaldata on the at least one two-dimensional plane. For example, the system150 may colorize the optical data in the two-dimensional plane toresemble the path side imagery simulating the captured image of theimage bubble.

The system 150 rotates the path side data based on the segment selectedfrom the map data. The path side data or imagery is rotated to becoplanar with the map data or map image. In other words, thetwo-dimensional plane including path side data, which is definedperpendicular to the map, is rotated and overlaid on the map. Forexample, the map may include a graphical model of roads (such as FakeRd., Green St., Blue Ave., Airport Rd., 2^(nd) St., Main St., 1^(st)St., Maple Ave, and/or other roads or paths), points of interest (suchas an airport, a park, or a building), and other geographic or mapfeatures. The map is or is not photo/video imagery data. For example,the map is a vector-based (e.g., SVG format), raster or pixel based,tile-based, or other type of graphical map model or representation. Forexample, the roads in the map are displayed based on map data, such as aroad segment and corresponding nodes. Graphical representations of themap are generated and/or displayed based on such map data. The map maybe displayed in a 2D (such as a bird's eye view), 2.5D (such as aperspective view), or 3D view.

The system 150 outputs the path side data and a portion of the map datato a display or an external device. Alternatively or in addition, thesystem 150 may store the image file including the map and the pointsprojected from the optical data and having pixel values defined by theimage bubble in the database 130. The pixel values may include one ormore of a color, a hue, a brightness, a luminance, and an intensity ofthe pixel.

FIG. 2 illustrates the projection of optical data 201 on atwo-dimensional plane 203. The optical data 201 may be generated by anoptical distancing system that employs one or more lasers to collectdata points representing an area, such as an area about a road 205,walkway, or other pathway. Software generates the optical data based onthe measured distance, the location of the optical distancing system,which may be on a moving platform such as a car (or Segway, animal,pedestrian, or other vehicle or object), and the angle of the laser.Optical distancing systems include LIDAR, a stereoscopic camera, a timeof flight infrared camera, and a structured light device.

A LIDAR device collects and gathers data points in a point cloud inwhich each data point corresponds to a local coordinate, such as (x, y,z). The LIDAR data may be a grayscale point cloud including an intensityfor each data point, ranging from 0 for black to 255 for white or viceversa. The point cloud may be stored in ASCII or LIDAR exchange format.The one or more lasers may be in a near infrared spectrum (such as about700 nm to about 5000 nm or about 800 nm to about 2500 nm) or other lightspectrum.

The view on the left side of FIG. 2 illustrates the optical data in thegeographically referenced three-dimensional space. The orthogonal lines207 show the projection of the optical data onto the two-dimensionalplane 203. The projection may be an orthographic projection or any othertype of projection for representing three-dimensional points in twodimensions. The orthographic projection is defined as any parallelprojection, including projection lines (e.g., orthogonal lines 207)perpendicular to the projection plane (e.g., two-dimensional plane 203),which preserves a relationship between the original points. Theorthographic projection may include exactly one translation and onelinear transformation for each point. The view on the right side of FIG.3 illustrates the two-dimensional plane 203 including the projectedoptical data 209.

FIG. 3 illustrates a view correlating the optical data 201 to an imagebubble 210. The geo-referenced three-dimensional space of the opticaldata 201 is shown on a real world scene including a building 213 andtrees 211. The optical data 201 is aligned with the Cartesian space ofthe image bubble 210 as shown by Cartesian axes 212. The alignment maybe calculated using an orientation quaternion. An orientation quaternionmay be defined as a four element mathematical notation for describing arotation. The orientation quaternion may describe the rotation from thegeo-referenced three-dimensional space to the Cartesian space or viceversa.

FIG. 4 illustrates another view correlating the optical data 201 to theimage bubble 210. Individual portions of the optical data 201 correspondto pixels of the image bubble 210. For example, a first pixel 221 a inthe image bubble 210 near the top of building 213 corresponds to a firstoptical datum 201 a, a second pixel 221 b in the image bubble 210 on thetree 211 corresponds to a second optical datum 201 b, and a third pixel221 c in the image bubble 210 near the door of building 213 correspondsto a third optical datum 201 c.

One or more pixel values from the image bubble 210 are applied to one ormore data points in the optical data 201. In some implementations, thereis a one to one ratio in the application of pixel values from the imagebubble 210. For example, a pixel value from the first pixel 221 a isstored in the two-dimensional plane 203 in association with the firstoptical datum 201 a, a pixel value from the second pixel 221 b is storedin the two-dimensional plane 203 in association with the second opticaldatum 201 b, and a pixel value from the third pixel 221 c is stored inthe two-dimensional plane 203 in association with the third opticaldatum 201 c. In other implementations, a group of pixels in the imagebubble 210 may be applied to one data point in the optical data 201. Thepixel values of the group of pixels may be averaged to select the newvalue of the data point in the optical data 201. Alternatively, themedian pixel value of the group of pixels may be used or the pixelvalues may be weighted according to the location (e.g., the weight ofeach pixel may be inversely proportional to the distance from the centerof the group of pixels).

In other implementations, a group of pixels in the image bubble 210 maybe applied to an area surrounding a data point in the optical data 201.The group of pixels in the image bubble 210 may be referred to as animage splat. A larger section of image data is cut from image bubble 210and applied to the two-dimensional plane. The size of the section ofimage data may be measured in pixels. The size of the section of imagedata may be a function of the location of the data point in the opticaldata 201. Data points farther from the perspective of the user may beassociated with a smaller number of pixels from the image bubble 210 anddata points closer to the perspective of the user may be associated witha larger number of pixels from the image bubble 210. In another example,size of the section of image data may be a function of the length oforthogonal line 207 associated with the particular data point in theoptical data.

In another example, a single pixel value from the image bubble 210 maybe applied to multiple data points in the optical data 201. Across thetwo-dimensional plane any combination of these algorithms may be used.The selection of the algorithm may be dependent on the density of theoptical data 201, which may vary significantly, as shown by FIG. 3.Interpolation may be used to obtain a continuous set of optical data201.

In another implementation, the color may be generated entirely based onthe depth information without using the image pixels. For example, thecolor may be based on depth change to indicate edge outlines. Thismethod is used to generate a gray scale silhouette or line based streetside view scene. In another mode, the edge color may be mixed with theimage pixels to create an image based street side scene with enhancededges.

The pixel values may specify color. The pixel values may include a redvalue, a green value, and a blue value, referred to collectively as aRGB value. Alternatively, the pixel values may be grayscale. Theresulting “image” in the two-dimensional plane 203 is a collection ofcolorized optical data. In addition or in the alternative, the pixelvalues may specify a hue, a brightness, a luminance, and/or an intensityof the pixel. Hue may be defined as the degree to which a stimulus canbe described as similar to or different from the unique hues (red, blue,green, yellow). Brightness may be defined as the degree to which thepixel appears to be radiating or reflecting light. Likewise, luminancemay be defined as a measurement of the luminous intensity per unit areaof light (e.g., candela per square meter). Intensity may be defined aslightness, amplitude, color value, color tone, or any description thatrelates to the subjective perception of human being on thelightness-darkness axis as described by a color model. Any of the abovequantities may be described by the Lab color model, the hue, saturation,lightness (HSL) color model, or the hue, saturation, and value (HSV)color model.

FIG. 5 illustrates mapping the optical data 201 to an image bubble 210.The process may inversely involve the mapping of spherical coordinatesof the image bubble 210 to the geographical coordinate space of theoptical data 201. Through geometry, an example point (P_(3D)) in theoptical data 201 may be referenced as a vector extending from the centerpoint (O) of the image bubble 210 having radius (r). The normalizedvector includes an X-coordinate (V_(x)) a Y-coordinate (V_(y)), and aZ-coordinate (V_(z)). The vector, or a line extending in the directionof the vector, intersects the image bubble 210. The pixel values at ornear the intersection of the vector and the image bubble 210 are storedas the two-dimensional plane 203. Equations 1 and 2 provide an examplecalculation for mapping the example point P_(3D) to the sphericalcoordinates (θ, φ) from which the pixel value is determined. Thespherical coordinates may be converted to image pixel coordinates basedon the image dimensions (i.e., the number of pixels in the image).

φ=arctan(−Vy/Vx)  Eq. 1

θ=arcsin(−Vz)  Eq. 2

FIG. 6 illustrates a map 230 including path side data or imagery in anorthographic projection. The path side data include a first portion 231on one side of the path and a second portion 232 on the other side ofthe path. Each portion may be projected onto a two-dimensional plane anddetermined according to respective panoramic images as discussed above.Further, each portion is rotated approximately 90 degrees or otherwiseoverlaid on the map 230 adjacent to the path segment 233. The firstportion 231 may coincide with a geographical direction, which determinesplacement on the map 230. For example, the first portion 231 is shownadjacent to the path segment 233 on the north side of the path segment.Similarly, the second portion 232 may coincide with the south side ofthe path segment 233 and be placed adjacent and to the south of thesegment 233 in the map 230.

FIGS. 7A and 7B illustrate an example rotation of the path side data. Anoriginal point P is a three-dimensional optically measured point(optical data). The point P may represent a building or another object.The road or path shown in FIGS. 7A and 7B runs into the page and theview of FIGS. 7A and 7B is horizontal along the road or path.

Rotation point F illustrates the axis about which the path side data isrotated or folded. The rotation point F may be defined based on the roadwidth W. The rotation point F may be calculated as F=C+WN, where N is aunit vector normal to the road or path direction. Alternatively, therotation point F may be calculated based on the optical data to identifya feature such as a curb or building façade.

The path side data may be rotated using a rotation matrix, a rotationvector, or a rotation quaternion. FIG. 7B illustrates the rotation ofpoint P using a rotation matrix M_(R) to P_(rotated). The rotationmatrix M_(R) may be defined by a rotation axis tangent to the road orpath direction. As shown by Equation 3, the rotated point of anythree-dimensional point of the path side data is based on the rotationpoint F and the rotation matrix M_(R).

P _(rotated) =M _(R)*(P−F)+F  Eq. 3

Equation 3 further illustrates that the rotation about rotation point Fis performed by first shifting rotation point P to the origin (P−F), andsubsequently rotating the shifted point according to the rotation matrixM_(R), and finally reversing or undoing the shift with the term (+F).

For a rotation of 90 degrees to align the path side data to be coplanarwith the map data, the rotation matrix M_(R) may be:

$M_{R} = \begin{bmatrix}0 & {- 1} & 0 \\1 & 0 & 0 \\0 & 0 & 1\end{bmatrix}$

Additionally, general rotation matrices for rotating the path side dataany combination of three axial angle rotations θ_(x), θ_(y), θ_(z)include a plurality of rotation matrices R_(x), R_(y), and R_(z). Theresult of successive rotations may be is any order such as R=Rx*Ry*Rz

${{Rx} = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; \theta_{x}} & {\sin \; \theta_{x}} \\0 & {{- \sin}\; \theta_{x}} & {\cos \; \theta_{x}}\end{bmatrix}},{{Ry} = \begin{bmatrix}{\cos \; \theta_{y}} & 0 & {{- \sin}\; \theta_{y}} \\0 & 1 & 0 \\{\sin \; \theta_{y}} & 0 & {\cos \; \theta_{y}}\end{bmatrix}},{{Rz} = {\begin{bmatrix}{\cos \; \theta_{z}} & {\sin \; \theta_{z}} & 0 \\{{- \sin}\; \theta_{z}} & {\cos \; \theta_{z}} & 0 \\0 & 0 & 1\end{bmatrix}.}}$

The placement on the map 230 of the first portion 231 and the secondportion 232 may be dependent on the travel of the vehicle 236, whichincludes the user device 100. For example, the first portion 231 may beshown to coincide with the right side of the vehicle 236 and the secondportion 232 may be shown to coincide with the left side of the vehicle236. As shown in FIG. 6, the vehicle 236 travels from right to left. Ifthe vehicle 236 travels from left to right, the first portion 231coincides with the left side of the vehicle 236, and the second portion232 coincides with the right side of the vehicle 236. In other words,the placement of the path side data may also be dependent on thelocation and/or movement of the user device 100.

The map 230 may also include additional path segments, such as pathsegment 234. In one implementation, additional path side data or imagerycorresponding to path segment 234 may be included on map 230. The map230 may also include point of interest data or icons 235. Among otherthings, the point of interest data may indicate shopping, restaurants,lodging, hospitals, and public transportation.

The computing resources necessary for generating a map including pathside data or imagery may be divided between the server 120 and the userdevice 100. In some embodiments, the server 120 performs a majority ofthe processing (“server-based embodiments”). In other embodiments, theuser device 100 performs a majority of the processing (“userdevice-based embodiments”).

For navigation related applications, the user device 100 may generate alocation according to the geographic location of the user device 100.The location may be generated using positioning circuitry including oneor more of a global navigation satellite system based on a satellitesignal, a triangulation system that utilizes one or more terrestrialcommunication signals, a inertial position system based on sensors suchas gyroscopes or accelerometers, and/or a dead reckoning system based ona previously known position. The positioning circuitry may alsodetermine an orientation using any of the above systems and/or amagnetic sensor such as a compass. The orientation and the locationallow the appropriate depthmap and panoramic image to be selected basedon the perspective of the user at the current geographic location of theuser device 100.

The network 110 may include any combination of a cellular network, theInternet, or a local computer network. For example, the user device 100may communicate with the network 110 wirelessly though protocols knownas Wi-Fi, the protocols defined by the IEEE 802.11 standards, theprotocols defined by the Bluetooth standards, or other protocols.Alternatively or in addition, the user device 100 may communicate withthe network 110 wirelessly as a cellular network such as a Global Systemfor Mobile Communication (GSM) or other third generation (3G) or fourthgeneration (4G) networks.

The optical distancing system may be a LIDAR device, a stereoscopiccamera, a time of flight camera, a structured light device, or anotherdevice. In the example of a LIDAR device, one or more (e.g., 1, 2, 10)lasers rotate in a plane. The optical distancing system may be coupledwith an inertial measurement unit (IMU) to associate the opticaldistance data with the geo-located position of the optical distancingsystem.

The optical distancing system measures the distance from the opticaldistancing system to the various objects. In another example, thestructured light device emits light onto objects and a camera capturesimages the structured light to determine distance based on the shape ofthe light in the captured images. The structured light may be infraredlight or another light spectrum not visible to humans.

FIG. 8 illustrates another map 240 including path side data or imagery.The path side data or imagery may also be divided into a first portion241 on one side of the path segment 243 and a second portion 242 onanother side of the path segment 243. The map 240 may also includebuilding footprints 247. The building footprints 247 are substantiallyaligned with the buildings shown in the path side data or imagery. Thevehicle 246 is nearing the node 245 which corresponds to theintersection of path segment 243 and path segment 244.

In navigation applications, the vehicle 246 may include the user device100. When the vehicle 246 turns from path segment 243 onto path segment244, the path side data for the first portion 241 and the second portion242 is removed from the map 240. Path side data that corresponds to thepath segment 244 is added to the map 240.

FIG. 9 illustrates a detailed view of the server 120 of FIG. 1. Theserver 120 includes a server controller or circuitry 500, a memory 501,and a communication interface 505. The database 130 may be external orinternal to the server 120. In the server-based embodiments, the server120 is an apparatus for generating a map including path side data orimagery.

FIG. 10 illustrates a detailed view of the user device of FIG. 1. In theuser device-based embodiments, the user device 100 is an apparatus forgenerating a map including path side data or imagery. The user device100 includes a user device controller or circuitry 600, a memory 601, acommunication interface 605, and position circuitry 607. The user device100 may also include a user input device 603 and a display 611.

The server processor 500 or another computing device is a processorconfigured to rotate the path side data based on a segment selected fromthe map data and generate an image including the path side data and atleast a portion of the map data. The server processor 500 may calculatethe rotation using a rotation matrix or a rotation quaternion. Innavigation related applications, the segment is selected based on thelocation of the user device 100. Otherwise, the segment may be selectedby the user or the routing algorithm. The communication interface 605 isconfigured to receive map data from a map database. The map dataincludes a plurality of nodes and a plurality of segments. The database130 is a memory configured to store path side data referenced tothree-dimensional geographic coordinates.

The server processor 500 is also configured to project optical datathrough geometrical calculations. As shown in FIG. 2, the optical datais projected onto two-dimensional plane 203 along an orthogonal line207. The server processor 500 generates the path side data by defining apixel value for a point of the optical data on the two-dimensional 203plane according to image data from at least one panoramic image. Inother words, the server controller 500 utilizes the perspective of theimage bubble and the geo-referenced location of the optical data toselect a color or other graphical attribute for the corresponding pointin the two-dimensional plane.

The server controller 500 may store the two-dimensional plane includingthe colorized optical data in database 130 in the format of an imagefile. The format may be a bit map, a portable network graphics (PNG)file, a lossy compression graphics file, or other format. Thetwo-dimensional plane including the colorized optical data may beassigned to a navigation link or segment, point-of-interest (POI),location reference object (LRO), node, and/or another type of route/mapdata.

The route data may be calculated from an origin point to a destinationpoint by the server controller 500 or externally. The route dataincludes nodes, which represent intersections or intersection points,and links or path segments, which connect the nodes. The route data maybe stored in a geographic database (e.g., database 130) in a spatialformat (e.g., Oracle spatial format), which is maintained by the mapdeveloper and compiled into a delivery format (e.g., geographical datafile (GDF) format).

The route data may be defined by a routing algorithm based on a Dijkstramethod, an A-star algorithm or search, and/or other route exploration orcalculation algorithms. Various aspects, such as distance, non-navigableareas, costs, and/or restrictions, are considered to determine anoptimum route. The routing algorithm may be specific to pedestrianrouting. The routing algorithm may rank links or segments according tosuitability for traversal by pedestrians. For example, links or segmentsmay be classified according to a plurality of pedestrian modes,including walking, bicycle, and wheelchair.

The apparatus for generating a map including path side data or imagerymay operate with or without position data for the current location ofthe user device 100. When the position data is used in the server-basedembodiments, the position circuitry 607 determines a current geographicposition of the user device 100 and the communication interface 605sends the current geographic position to the server 120. When theposition data is used in the user-device based embodiments, the positioncircuitry 607 determines location data including the position and/or theorientation of the user device 100. The location data may be generatedusing one or more of a global navigation satellite system based on asatellite signal (such as Global Positioning System (GPS), the RussianGLONASS or European Galileo), a triangulation system that utilizes oneor more terrestrial communication signals, a inertial position systembased on relative position sensors such as gyroscopes, accelerometers,and altimeters, and/or a dead reckoning system based on a previouslyknown position. The orientation may be determined using any of the abovesystems and/or a magnetic sensor such as a compass or athree-dimensional magnetic sensor array. Magnetic sensors determine thedirection and or strength of a magnetic field and can be used todetermine heading or orientation. Inertial sensors such asaccelerometers and gyroscopes measure acceleration, which can be used tocalculate position, orientation, and velocity (direction and speed ofmovement) of the user device 100. The location and/or orientation datamay be used to select the navigation segment associated with thetwo-dimensional plane from the database 130.

In the user device-based embodiments, the memory 601 is configured tostore the optical data measured in three-dimensional geographiccoordinates and image data from at least one image bubble. The devicecontroller 600 may be configured to project the optical data onto atleast one predefined two-dimensional plane and calculate a pixel valuefor a point of the optical data on the at least one predefinedtwo-dimensional plane according to the image data using the algorithmsdiscussed above. The device controller 600 is configured apply arotation matrix or rotation quaternion to the path side data to rotatethe path side data based on a segment selected from the map data andgenerate an image including the path side data and at least a portion ofthe map data.

The predefined two-dimensional plane is converted to a format similar toan image. The image may be stored in memory 601 and displayed on display611. The device controller 600 may generate an image file from thepredefined two-dimensional plane and associate the image file with anavigation segment (the navigation segment may refer to datarepresenting a road or path segment in a map database, a portion of anavigation route, and or any other geographical/navigation/map area orobject). The user input device 603 is configured to receive a commandindicative of the navigation segment. The command could be in responseto the calculation of a route or route data. Alternatively, the commandcould be in response to selection of an address, a point of interest, aselection on a digital map, or manual selection of an image bubble.

The display 611 is configured to display street side imagery from theimage file in response to the command. The display 611 may be combinedwith the user input device 603 as a touch screen, which may capacitiveor resistive. In addition, the user input device 603 may include one ormore buttons, keypad, keyboard, mouse, stylist pen, trackball, rockerswitch, touch pad, voice recognition circuit, or other device orcomponent for inputting data to the navigation device 100. The display611 may be a liquid crystal display (LCD) panel, light emitting diode(LED) screen, thin film transistor screen, or another type of display.

The user device controller 600 or server controller 500 may include ageneral processor, digital signal processor, an application specificintegrated circuit (ASIC), field programmable gate array, analogcircuit, digital circuit, combinations thereof, or other now known orlater developed processor. The user device controller 600 or servercontroller 500 may be a single device or combinations of devices, suchas associated with a network, distributed processing, or cloudcomputing.

The memories 501, 601 may be a volatile memory or a non-volatile memory.The memory 501, 601 may include one or more of a read only memory (ROM),random access memory (RAM), a flash memory, an electronic erasableprogram read only memory (EEPROM), or other type of memory. The memory501, 601 may be removable from the user device 100, such as a securedigital (SD) memory card.

The communication interfaces 505, 605 may include any operableconnection. An operable connection may be one in which signals, physicalcommunications, and/or logical communications may be sent and/orreceived. An operable connection may include a physical interface, anelectrical interface, and/or a data interface. The communicationinterfaces 505, 605 provides for wireless and/or wired communications inany now known or later developed format.

In an alternative embodiment, the user device 100 may omit the positioncircuitry 607 or use of the position circuitry 607. In this alternativeembodiment, the user device 100 may be a personal computer, whichencompasses laptops and other mobile processing platforms.

FIG. 11 illustrates a map 250 including silhouettes 251 of path sideimagery. The silhouettes are not colorized or otherwise defined fromassociated panoramic images. The silhouettes 251 are generated byassigning all projected optical data to the same pixel value. Theoptical data may be filtered to remove extraneous points. For example,the server processor 500 may be configured to generate the path sidedata by defining a pixel value for points of the optical data projectedaccording to a predetermined pixel value. The pixel value may representblack.

FIG. 12 illustrates a map 260 including outlines 261 of path sideimagery. The outlines 261 are not colorized or otherwise defined fromassociated panoramic images. Instead, the outlines 261 are calculated byfiltering the optical data. For example, the trends in the optical dataare analyzed to determine when an object ends or begins. A steep changein the depth in a series of optical data indicates the beginning or endof an object. In one example algorithm, the depth value for one point issubtracted from a subsequent point and the difference is compared with athreshold value. When the difference exceeds the threshold value, thecontroller sets a pixel value at that point to a predetermined value. Asthis process is repeated across the depthmap, outlines are drawn aroundbuildings and other objects.

FIG. 13 illustrates a map 270 including optical data 271 of path sideimagery. The optical data may be generated from a light detection andranging (LIDAR) device, a stereoscopic camera, or a structured lightdevice. The optical data 271 is not colorized or otherwise defined fromassociated panoramic images. Features of objects are visible in theoptical data because of the reflectivity of the object. The intensityvalues of the optical depend on the reflectivity of the objects. Thepath side data based on reflectivity has no natural lighting, noshadows, and may be collected in complete darkness. The path side databased on reflectivity highlights the architectural details of thebuilding facade. The server controller 500 or device controller 600 maybe configured to filter the optical data according to the reflectivityof the optical data and generate the path side data based on thefiltered optical data.

FIG. 14 illustrates a map 280 including a perspective view 281 of pathside imagery. To create the perspective view 281, the controllerutilizes the depth values of the optical data. The optical data isprojected onto a two-dimensional place, as discussed above, butprojected using a standard perspective transformation instead of anorthographic projection. The perspective transformation draws objectssmaller as the depth value is higher until reaching a vanishing pointconvergence maximum. The controller omits points with a depth valuegreater than the vanishing point convergence maximum. This provides athree-dimensional effect according to the depth values of the opticaldata.

The perspective view 281 may be constructed from a series ofpredetermined images. Each of the series of predetermined images isassociated with a range of locations. For example, each image may beassociated with one city block or 50 meters of a pathway. Seams betweenthe images may be aligned based on objects in the foreground of theimages.

The system 150 may project the optical data onto a plurality ofpredefined two-dimensional planes. The plurality of predefinedtwo-dimensional planes may be defined by a distance range. For example,optical data from the center point to the curb of the street may besnapped to a first plane, optical data from the curb to just before thebuilding facade may be snapped to a second plane, and optical data fromthe building facade and optical data farther from the center point maybe snapped to a third plane. The system 150 may store an image file foreach plane in the database 130. Each image file includes pixel valuesdefined by the image bubble.

FIG. 15 illustrates the projection of the optical data 301 on aplurality of two-dimensional planes 303 a-c. The view on the left sideof FIG. 15 illustrates the optical data 301 in the geographicallyreferenced three-dimensional space. The plurality of two-dimensionalplanes 303 a-c may be defined in a variety of algorithms such as adistance algorithm, a statistical algorithm, an image processingalgorithm, or a database algorithm. The orthogonal lines 307 a-c showthe projection of the optical data onto the two-dimensional planes 303a-c. The view on the right side of FIG. 15 illustrates thetwo-dimensional planes 303 a-c including the projected optical data 309.

In the distance algorithm, the plurality of two-dimensional planes 303a-c are defined based on distance. Any number of planes may be used. Thedistance may be measured from the collection point of the optical data201 and/or the collection point of the image bubble 210. The distancemay be measured from another reference point. The first plane 303 a maybe associated with a first threshold distance (e.g., 1 m, 5 m, 10 m).Points within the first threshold distance are projected onto the firstplane 303 a. The second plane 303 b may be associated with a secondthreshold distance (e.g., 2 m, 5 m, 10 m). Points farther from thecenter point than the first threshold distance but within the secondthreshold distance are projected onto the second plane 303 b. The thirdplane 303 c may be associated with a third threshold distance (e.g., 5m, 10 m, 20 m). Points farther from the center point than the thirdthreshold distance are projected on the third plane 303 c. Theprojection of the optical data 201 to the two-dimensional planes 303 a-cmay be according to the processes discussed above with respect to FIG.2.

The statistical algorithm may define the plurality of two-dimensionalplanes 303 a-c according to the values of the optical data 201. Forexample, the optical data 201 may be divided into N groups of data. Forexample, if N=2, the optical data 201 is divided in half according todistance. A first half of the optical data 201 is projected onto a firstplane, and a second half of the optical data 201 is projected onto asecond plane. N may be any integer value and an amount of data equal to100/N percent of the total optical data 201 is projected on each plane.Alternatively, the optical data 201 could be divided according tostandard deviation, variance, or the rate of change of distance.

The image processing algorithm may define the plurality oftwo-dimensional planes 303 a-c according to trends in the optical data201. The trends may identify navigation features in the optical data.The navigation features may include one or more of a road surface, acurb, a sign, a parked car, or a building facade. For example, theoptical data 201 may be analyzed to determine the location of a curb, abuilding facade, or another object in the data. The image processingalgorithm divides the optical data 201 such that points from the roadsurface to the curb are snapped to the first plane 303 a, points fromthe curb to just before the buildings are snapped to the second plane303 b, and points from the building facade and beyond are snapped to thethird plane 303 c.

The database algorithm may define the plurality of two-dimensionalplanes 303 a-c according to a database of physical features, which maybe included in the map database or geographic database 130. The physicalfeatures may be building footprints, the number of lanes or dimensionsof the pathway, or any known reference points that define the locationof the building facade. As discussed above, the path side data in theplurality of two-dimensional planes 303 a-c are rotated to be alignedwith map data. The rotation may be performed using a rotation matrix orrotation quaternion.

All of the points in the path side imagery are snapped, projected, orcompressed into the discrete layers. Each layer has the same lengthdimension in the path side imagery but corresponds to differentgeographical distances in the image bubbles and, accordingly, in thereal world. Therefore, as the path side imagery is scrolled to view theentire street, the layers may scroll at different rates. The differentrates may be proportional to the ratio of the first threshold distanceand the second threshold distance.

Further, the controller may assign transparency values to each point inthe path side imagery. The transparency of pixels in one layer defineshow much of a subsequent layer may be seen. The combined transparency ofthe layers defines how much of the map data behind the path side imageryis visible. Alternatively, the controller may calculate an opaquenessvalue for pixels of the path side imagery.

FIG. 16 illustrates a plurality of image bubbles 410 a-e correlated witha street side view. Because of the panoramic nature of the image bubbles410 a-e, successive image bubbles overlap. That is, a single point inthe real world, and accordingly in optical data 201, occurs in multipleimage bubbles 410 a-e. This principle is shown by building 411 in FIG.16. Each of image bubbles 410 a-e may provide a different perspective ofbuilding 411. Any of the algorithms for selecting the pixel values forthe predefined two-dimensional plane, as described above, may bemodified to include pixel values from a plurality of image bubbles. Thepixel values from the plurality of image bubbles may be averaged. Inother examples, pixel values from certain image bubbles may be ignored.

FIG. 17 illustrates an example obstruction 422 and multiple imagebubbles 420 a-c. The image bubbles 420 a-c may include any number ofimage bubbles such as three, five, or ten image bubbles spaced atdistance intervals (e.g., 1 m, 5 m, 10 m) or at time intervals of amoving camera. For a particular point 423 on the object 421, theassociated pixel value in image bubble 420 a is Color A, the associatedpixel value in image bubble 420 b is Color B, and the associated pixelvalue in image bubble 420 c is Color C. Color A and Color Cappropriately correspond to different perspectives of the particularpoint 423. However, Color B corresponds to obstruction 422 between theimage bubble 420 b and the object 421. If Color B was averaged orotherwise considered in the selection for the data point of thetwo-dimensional plane, the obstruction 422 would adversely affect theappearance of the street side image 401. Therefore, Color B may beignored and Color A and Color C may be averaged. The system 150 mayidentify that Color B was caused by an obstruction, or otherwiseerroneous, because Color B is an outlier. An outlier may be defined as apixel value far away from the other pixel values from other imagebubbles corresponding to the same point in the optical data 201. Forexample, any pixel value that is a predetermined number (e.g., 1, 2, 3)of standard deviations away from the mean pixel value may be consideredan outlier. Alternatively, any pixel value that differs from the meanpixel value by minimum variation may be considered an outlier. Forexample, the minimum variation may be 50 or 100 on a 0 to 255 scale forpixel values. Example outliers 422 may be caused by light poles, movingcars, pedestrians, trees, leaves, or any object that may be between thebuilding facade and the camera at some perspectives and not at others.

FIG. 18 illustrates a flowchart for generating a map with path/streetside imagery. Fewer, more, or different steps or acts may be provided,and a combination of steps may be provided. Also, the steps or acts maybe performed in the order as shown or in a different order. The methodis implemented by the system and/or devices described herein or bydifferent devices or systems. One or more steps or processes of themethod may be fully or partially automated (such as via a computer,processor, and/or algorithm).

At act S101, path side data referenced to three-dimensional geographiccoordinates is stored. The path side data may be derived from opticaldata collected by an optical distancing system. The optical data may bereceived from the optical distancing system or received from memory atthe controller, which alternately refers to either the server controller500, the device controller 600, or another controller or computingdevice. The optical data may be projected onto one or more predefinedtwo-dimensional planes. The controller may calculate pixel values forthe projected optical data based on a correlated panoramic image.Alternatively, the controller may calculate pixel values for theprojected optical data based on an analysis of the optical data.

At act S103, the controller receives map data from a map database. Themap data includes nodes and segments corresponding to paths andintersections. The map data may be stored in a spatial format or adelivery format (e.g., GDF format). The controller may be configured toconvert the map data to an image. At S105, the controller selects asegment from the map data. In some embodiments, the segment is selectedbased on a detected position of the user device 100. In otherembodiments, the segment is selected based on a routing algorithm or auser input.

At S107, the controller rotates the path side data based on the segment.The path side data may be rotated about the location of the segment.Alternatively, the path side data may be rotated about another locationsuch as the edge of the path or a curb. The edge of the path or a curbmay be determined by analyzed the optical data. The path side data maybe rotated using a rotation matrix, a rotation vector, or a rotationquaternion. The rotation aligns the path side data with the map data.

At S109, the controller outputs the path side data and a portion of themap data. The path side data and the map data may be combined in animage that is displayed to a user. The image may be stored as an imagefile. The image file may be output from the server 120 to be stored inthe database 130 or transmitted to the user device 100. The image filemay be output from the user device 100 to be shown on display 611. Inaddition, the controller may assign the at least one image file to anavigation link or segment or other map/navigation object, such as anode or POI. The at least one image file and assignment may be stored inthe database 130. In the case of multiple two-dimensional planes, the atleast one image file may include a plurality of image files such thateach image file corresponds to one of the plurality of predefinedtwo-dimensional planes.

The controller may output a subset of the plurality of image fileseither to a display 611 in the user device-based embodiments or from theserver 120 to the user device 100 in the server-based embodiments. Thesubset may be selected based on a configuration setting defined by arange of distance values. The subset may be selected based on ananalysis of the optical data. The subset may be selected based on anavigation feature detected by the controller in the optical data. Thenavigation feature may be a road surface, a curb, a building facade, orother man-made or natural geographic feature.

FIG. 19 illustrates curved planes as an alternative to the Euclideanplanes of FIG. 15. In the alternative to the two-dimensional planes 303a-c in FIG. 15, shown as planar in the Cartesian coordinates, thetwo-dimensional planes 303 a-c may be curved planes which are planar inspherical coordinates. The radius of the curved planes may be determinedby the curvature of an associated segment or pathway. Alternatively andas shown in FIG. 19, each of the two-dimensional planes 801 a-b may becurved at multiple locations having varying radii of curvature.

In one example, the curved surfaced are defined by a set of parametricequations. In other examples, the curves may be defined according to theoptical data 803 as defined by the distance algorithm, the statisticalalgorithm, the image processing algorithm, or the database algorithm, asdiscussed above. The lines 805 are orthogonal to the nearest point onthe surface of the two-dimensional planes 801 a-b. The orthogonal lines805 show the projection of the optical data onto the two-dimensionalplanes 801 a-b.

FIG. 20 illustrates an intermediate projection surface 811. Theintermediate surface 811 non-linearly maps the optical data 813, whichcorresponds to opposite side of an object (e.g., building 810) to thepath side data of the two-dimensional plane 812. The intermediatesurface 811 is a mathematic surface rather than a physical surface. Theintermediate surface 811 is defined to face multiple sides of thebuilding 810. The curvature of the intermediate surface 811 may becalculated by the controller from a building footprint or by analyzingthe optical data to identify the locations of buildings as discussedabove.

The optical data 813 is projected to the intermediate surface 811. Eachpoint of the optical data 813 is projected onto the closest location onthe intermediate surface 811, as shown by first projection lines 814.The data projected onto the intermediate surface 811 is then projectedonto the two-dimensional plane 812. The projection may be orthogonal.Alternatively, the projection may be based on the curvature of theintermediate surface 811 at each point, as shown by second projectionlines 815. In other words, the data is projected onto the intermediatesurface and then unwrapped into the two-dimensional plane 812. Thesecond projection lines 815 may be at an acute angle to (notperpendicular to) the intermediate surface 811 or the two-dimensionalplane 812 or both. The resulting path side data 816 is apseudo-perspective of the path side imagery (e.g., building 810). Thepseudo-perspective view of the path side imagery illustrates varioussides of the building 810 in a seamless image. The pseudo-perspectiveview of the path side imagery illustrates some of the sides ofbuildings, which may or may not be proportional to the actual sizes ofthe buildings. The sides of the buildings may include a sign, entrance,or other POI useful in the path side imagery. The pseudo-perspectiveview of the path side imagery may be colorized or otherwise modifiedaccording to any of the embodiments above.

FIGS. 21A and 21B illustrate another embodiment for generating a mapincluding path side imagery from a building footprint 820. The buildingfootprint 820 may be accessed from an external database. The controllermay be configured to calculate multiple folds of the building footprintto calculate path side data for the map. The multiple folds, as shown byangles a-d, unwrap or expand the three-dimensional building footprintbuilding to a two-dimensional image, which is rotated or overlaid on themap 821 as shown by FIG. 21B.

Alternatives

The embodiments above describe rotating the path side data to align withmap data in a two-dimensional map image. The rotation may be at anyangle. In addition, the rotation may be a variable bend or curve in athree-dimensional space. In the resulting three-dimensional image, thepath side data may form a U-shape or a saddle shape. In the U-shape thepath side data raises up on both sides of the path. In the saddle shapethe path side data raise up on one side of the path and goes down on theother. Bending or warping the path side data for display highlightsfeatures in the data.

An ‘inpainting’ algorithm may be used to fill spaces or holes betweenpoints. The inpainting algorithm may include blurring or diffusiontechniques. For example, the inpainting algorithm may determine a pixelvalue for a particular pixel by averaging pixel values of neighboringpixels. When the spaces or holes span more than one optical data pointor pixel, the inpainting algorithm may be repeated such that previouslyblurred/diffused pixels are averaged to determine a subsequent pixelvalue.

A hole filling algorithm and/or smoothing algorithm may be applied toresulting image planes to handle regions where the depth informationdensity is insufficient or where image information is obstructed in allimage views. The hole filling algorithm may interpolate pixel values fora selected pixel based on surrounding pixel values. The smoothingalgorithm may variably adjust pixel values based on surrounding pixelvalues.

The embodiments described above have been described substantially interms of street/path views. However, the embodiments herein may beapplied to indoor environments and other types of pathways, such asindoor pathways, hallways, and office settings. The indoor environmentmay be surveyed in three-dimensional geographic coordinates to generatethe optical data. The optical data may be created using any arbitraryviewpoint or perspective in the indoor environment. For example,shopping malls, pedways, and office buildings have pathways central tolocations or points of interest within the indoor environment. Theoptical data may include geo-coordinates such as latitude, longitude,and altitude for each of plurality of points based on an IMU and/or GPS,as described above.

The optical data may be projected onto at least one predefinedtwo-dimensional plane parallel to the pathway and/or the direction ofmovement of the optical distancing system as the optical data iscollected. The predefined two-dimensional plane may be selected toestimate a storefront or office facade along the pathway. Image datafrom at least one panoramic image or image bubble, collected by acamera, may be associated with the storefront or office facade andreferenced in another coordinate system. The image data of the imagebubble is correlated with the geo-referenced three-dimensional space ofthe optical data. The two-dimensional plane is colorized or otherwisemodified based on the image data. The data from the two-dimensionalplane is rotated to overlay a map of the indoor environment. Theresulting data is stored as an image file, which can be stored in a mapor geographic database to simulate the view of a pedestrian from thepathway.

The embodiments described above including multiple layers of street/pathside imagery include layers divided horizontally. Horizontal layers aredefined by distances in the horizontal direction. The embodimentsdescribed above could alternatively or additionally include verticallayers defined by distances in the vertical direction. For example, theoptical data may be projected onto a plurality of vertical planesdefined by a user or automatically. The user may define one or morevertical thresholds. The vertical distance may be measured from thecollection point of the optical data and/or the collection point of theimage bubble. Alternatively, the vertical planes may be definedautomatically. For example, the vertical planes may be selectedaccording to the values of the optical data such that the optical datais divided evenly according to distance. Each optical data point may beprojected onto the closest vertical plane. In another example, an imageprocessing algorithm may define the planes according to trends in theoptical data identifying navigation features in the optical data. Thenavigation features may include one or more of a building facade, astorefront, a doorway, a door, a window, or another feature.

The street/path side imagery is divided into a layer for each of thevertical planes. A first layer includes points with pixel valuesselected from one or more image bubbles according to the optical dataprojected onto a first plane. For example, the first layer may includeobjects closest vertically to the perspective of the street/path sideimagery. The second layer includes points with pixel values selectedfrom one or more image bubbles according to the optical data projectedonto a second plane. In an outdoor environment, the first layer mayinclude the lower building facade, and the second layer may include thetops of buildings. In an indoor environment, such as a shopping mall,the first layer may include storefronts on the first floor and thesecond layer may include storefronts on the second floor. In anotherexample, the plurality of planes may be spherically shaped and definedby a threshold radius from the collection point of the optical dataand/or the collection point of the image bubble.

The embodiments described above may be implemented using computerexecutable instructions stored in the memory 501, the memory 601, and/orother memory, which are non-transitory. The processors may executecomputer executable instructions. The computer executable instructionsmay be written in any computer language, such as C++, C#, Java, Pascal,Visual Basic, Perl, HyperText Markup Language (HTML), JavaScript,assembly language, extensible markup language (XML) and any combinationthereof.

The computer executable instructions may be logic encoded in one or moretangible media or one or more non-transitory tangible media forexecution by the processors. Logic encoded in one or more tangible mediafor execution may be defined as instructions that are executable by theprocessors and that are provided on the computer-readable storage media,memories, or a combination thereof. Instructions for instructing anetwork device may be stored on any logic. As used herein, “logic”,includes but is not limited to hardware, firmware, software in executionon a machine, and/or combinations of each to perform a function(s) or anaction(s), and/or to cause a function or action from another logic,method, and/or system. Logic may include, for example, a softwarecontrolled microprocessor, an ASIC, an analog circuit, a digitalcircuit, a programmed logic device, and a memory device containinginstructions.

The computer readable instructions may be stored on any non-transitorycomputer readable medium. A non-transitory computer readable medium mayinclude, but are not limited to, a floppy disk, a hard disk, an ASIC, acompact disk, other optical medium, a random access memory (RAM), a readonly memory (ROM), a memory chip or card, a memory stick, and othermedia from which a computer, a processor or other electronic device canread.

As used herein, the phrases “in communication” and “couple” are definedto mean directly connected to or indirectly connected through one ormore intermediate components. Such intermediate components may includeboth hardware and software based components.

As used in this application, the term ‘circuitry’ refers to all of thefollowing: (a) hardware-only circuit implementations (such asimplementations in only analog and/or digital circuitry) and (b) tocombinations of circuits and software (and/or firmware), such as (asapplicable): (i) to a combination of processor(s) or (ii) to portions ofprocessor(s)/software (including digital signal processor(s)), software,and memory(ies) that work together to cause an apparatus, such as amobile phone or server, to perform various functions) and (c) tocircuits, such as a microprocessor(s) or a portion of amicroprocessor(s), that require software or firmware for operation, evenif the software or firmware is not physically present. This definitionof ‘circuitry’ applies to all uses of this term in this application,including in any claims. As a further example, as used in thisapplication, the term “circuitry” would also cover an implementation ofmerely a processor (or multiple processors) or portion of a processorand its (or their) accompanying software and/or firmware. The term“circuitry” would also cover, for example and if applicable to theparticular claim element, a baseband integrated circuit or applicationsprocessor integrated circuit for a mobile phone or a similar integratedcircuit in server, a cellular network device, or other network device.

The embodiments described above may be combined with systems and methodsfor generating path side imagery as described in copending application“PATH SIDE IMAGERY” by James D. Lynch, filed Dec. 30, 2011 (AttorneyDocket No. N0408US), which is incorporated by reference in its entirety.

Various embodiments described herein can be used alone or in combinationwith one another. The foregoing detailed description has described onlya few of the many possible implementations of the present invention. Itis intended that the foregoing detailed description be regarded asillustrative rather than limiting and that it is understood that thefollowing claims including all equivalents are intended to define thescope of the invention.

1. A method comprising: receiving a depthmap of optical distance datacorresponding to a pathway; receiving image data from at least onepanoramic image corresponding to a view from the pathway; correlatingthe optical distance data and the image data; defining pixel values fora range of points of the optical distance data according to a section ofthe image data; and rotating the pixel values correlated with theoptical distance data to a map plane using the depthmap.
 2. The methodof claim 1, further comprising: displaying a map and the pixel values inthe map plane.
 3. The method of claim 1, wherein the pixels are rotatedabout a path segment for the pathway.
 4. The method of claim 3, furthercomprising: accessing map data for the map plane from a map database,wherein the map data includes the path segment.
 5. The method of claim1, further comprising: averaging the pixel values for the section of theimage data; and defining a new value for the optical data based on theaveraged pixel values.
 6. The method of claim 1, further comprising:weighting the pixel values according to distance; and defining a newvalue for the optical data based on the weighted pixel values.
 7. Themethod claim 1, wherein the optical distance data is collected by alight detection and ranging (LIDAR) system.
 8. The method of claim 1,further comprising: filtering the optical distance data according toreflectivity of the optical distance data.
 9. The method of claim 1,further comprising: dividing the optical distance data into a pluralityof layers defined by distance to the path segment.
 10. The method ofclaim 9, further comprising: selecting at least one of the plurality oflayers according to a user input; and displaying the at least one of theplurality of layers and a portion of the map data.
 11. An apparatuscomprising: at least one processor; and at least one memory includingcomputer program code for one or more programs; the at least one memoryand the computer program code configured to, with the at least oneprocessor, cause the apparatus to at least perform: receive a depthmapof optical distance data corresponding to a road segment; receive imagedata from at least one panoramic image corresponding to a view from theroad segment; colorize the optical distance data based on the imagedata; and rotate the colorized optical distance data to a map planeincluding the road segment.
 12. The apparatus of claim 11, the at leastone memory and the computer program code configured to, with the atleast one processor, cause the apparatus to: display a map and the pixelvalues in the map plane.
 13. The apparatus of claim 11, wherein thecolorized optical distance data are rotated about a path segment for thepathway.
 14. The apparatus of claim 11, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus to: average pixel values from the image data; anddefine colorized optical distance data based on the averaged pixelvalues.
 15. The apparatus of claim 11, the at least one memory and thecomputer program code configured to, with the at least one processor,cause the apparatus to: weight pixel values from the image dataaccording to distance; and define colorized optical distance data basedon the weighted pixel value.
 16. The apparatus of claim 11, wherein theoptical distance data is collected by a light detection and ranging(LIDAR) system.
 17. The apparatus of claim 11, the at least one memoryand the computer program code configured to, with the at least oneprocessor, cause the apparatus to: divide the optical distance data intoa plurality of layers defined by distance to the path segment.
 18. Theapparatus of claim 17, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto: select at least one of the plurality of layers according to a userinput; and display the at least one of the plurality of layers and aportion of the map data.
 19. The apparatus of claim 11, wherein thecolorized optical distance data is rotated using a rotation matrix. 20.A non-transitory computer readable medium having stored thereon acomputer program, the computer program comprising instructions to causea processor to: identify a depthmap of optical distance datacorresponding to a pathway; identify image data from at least onepanoramic image corresponding to a view from the pathway; correlate theoptical distance data and the image data; determine pixel values for theoptical distance data according to a section of the image data; andalign the pixel values correlated with the optical distance data to amap plane using the depthmap.